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City Year is an education-focused organization founded in 1988 dedicated to helping students and schools succeed. City 
Year partners with public schools in 26 urban, high-poverty communities across the United States and through interna- 
tional affiliates in the United Kingdom and Johannesburg, South Africa. Diverse teams of City Year AmeriCorps mem- 
bers provide student, classroom, and schoolwide support to help students stay in school and on track to graduate from 
high school, ready for college and career success. Aproud memberof the AmeriCorps national service network, City Year 
is made possible by support from the Corporation for National and Community Service, school district partnerships, and 
private philanthropy from corporations, foundations, and individuals. 

City Year AmeriCorps members are 17- to 24-year-olds who commit to one year of full-time service in elementary, middle, 
or high schools. Working on 7- to 18-person, school-based teams, City Year AmeriCorps members (or corps members) de- 
liver City Year's Whole School Whole Child (WSWC) model, which is a portfolio of services delivered in and outside of the 
classroom. Specifically, the model is intended to serve all students in a school, providing additional support to students at 
risk of dropping out based on their attendance, behavior, and course performance. City Year's services include whole class 
and targeted literacy and math tutoring, attendance and behavior coaching, social and emotional development support, 
and after-school programming that includes homework help, tutoring, and enrichment activities. 

The components of City Year's WSWC model include the following: 

■ School climate. Through positive school climate activities and mentoring activities, City Year corps members 
build strong relationships with students. They encourage students to come to school more often and make at- 
tending school an enjoyable experience that students can look forward to each day. Corps members also support 
teachers by helping with differentiation of instruction (e.g., working with students who need extra attention in 
the areas of attendance, behavior, and course performance). 

■ Literacy and math. Through one-on-one tutoring, small-group tutoring, and learning enrichment activities, City 
Year corps members create and support activities in schools to directly increase students' learning capacity and 
sense of academic efficacy. Corps members are present in the classroom and deliver interventions or academic 
coaching for a "focus list" of students, selected because of their grades, standardized test performance, English 
language ability, as well as their behavior and attendance history. Interventions may be delivered via pull-out 
or push-in. Corps members also provide whole class supports, enhancing curriculum delivery and guiding and 
encouraging students in their work. 

■ Attendance and behavior. For students on a focus list requiring attendance and behavior intervention, corps 
members provide regular coaching and build relationships with students both in and out of regular class time. 
Students receive formal attendance coaching at least weekly via a Check-in, Check-out goal-setting protocol, and 
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formal behavior coaching through a City Year-developed leadership development curriculum that is typically 
delivered during lunch time or advisory periods. 

■ After-school program (ASP). City Year corps members run an academic-focused after-school program that 

provides homework help, tutoring, and support for school-specific learning initiatives (such as blended learning). 
Enrichment lessons designed to support social-emotional development are also provided. 


City Year first piloted the WSWC model in 2007 and, by 2011-12, had achieved greater consistency in the implementation 
of WSWC, including the delivery of both academic and student engagement practices and the collection of student data, 
and was serving more than 150 schools across the elementary, middle and high school grade spans. This report focuses 
on investigating the impacts of City Year's WSWC model on school-level performance on state English Language Arts 
(ELA) and math assessments 1 in the 2011-12, 2012-2013, 2013-2014 school years. Our analyses of City Year's impact on 
school performance were guided by the following research questions: 

■ How do the whole school or grade wide outcomes across performance in English Language Arts and math of City 
Year schools compare to outcomes of other similar schools that do not partner with City Year? 


■ To what extent does the performance of schools change after partnering with City Year and in comparison to the 
performance of other similar schools that do not partner with City Year? 


■ What has City Year's whole school (whole grade) impact been on City Year schools' academic performance? 


■ What are examples of schools that have delivered outsized impacts on academic performance? 


■ What is the predicted impact on academic performance by having City Year in a school? 


■ How does City Year's impact on academic performance change overtime? How does it vary within a city (or even 
within a school)? How does it vary across the elementary, middle, and high school levels? 


This report begins with a summary of the study's key findings, a description of the study's methodology, and a brief expla- 
nation of the analyses we conducted. Next, we describe the characteristics of the City Year sites and their partner schools. 
We then present the findings for our analyses of City Year's impact on schools' ELA and math performance. 


1 

We tested other outcome measures such as improvement in graduation rates, improvement in attendance rates, and implementation scores. We 
elected to not report these models because model fit statistics and other tests of linear or logistic regression assumptions suggested fundamental 
issues with these models and, therefore, inaccurate estimates. 
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Key Findings 


■ Schools that had partnered with City Year (CY schools) were more likely to show improve- 
ment on state assessments compared with schools that shared similar demographic and 
performance characteristics but that had not partnered with City Year (non-CY schools). 
These differences were found for performance on state ELA assessments in 2011-12, in 
2012-13, and in 2013-14, and for state math assessments in 2011-12 and in 2012-13. 

■ CY schools were approximately two times more likely to improve on state ELA assess- 
ments in 201 1-12, in 2012-13, and in 2013-14, compared with non-CY schools. 

■ CY schools were approximately two times more likely in 201 1-12 and three times 
more likely in 201 2-13 to improve on state math assessments, compared with non- 
CY schools. We also found evidence that CY schools improved in math in 2013-14, 
although this result was not statistically significant. 

■ On average, CY schools in every City Year site were more likely to improve on state ELA and 
math assessments between 201 1-12 and 2012-13 than the comparison non-CY schools. 

■ CY schools with a higher ratio of corps members to students had a higher likelihood of 
seeing improved ELA and math performance on state assessments in 2012-13. 

■ In 2013-14, schools in high-implementing CY sites 1 were approximately two times more 
likely to improve on their state's ELA assessment, compared with schools in low-implement- 
ing CY sites and the comparison non-CY schools 2 . 


1 

We used implementation indicators taken from a City Year end-of-school year survey to construct several indices of fidelity of imple- 
mentation of the WSWC model and of specific aspects of the model, including the math and ELA components. 

2 

This result applies to the 2013-14 school year, the year for which data were available reqardinq school-level implementation of the 
WSWC model. 
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Study Methodology 


The following section describes how we obtained and structured the data and the decision rules we used for including vari- 
ables in the descriptive and impact analyses. Next, we describe the procedures we used to create a matched comparison 
group of schools. Finally, we describe the three types of analyses we conducted to address City Year's research questions, 
including descriptive, fixed-effects linear regression, and fixed-effects logistic regressions for both ELA and math outcomes. 
We conclude with a discussion of the study's limitations. 


Data Sources and Types 

For the analyses, we used data collected by City Year staff and data downloaded from state and district websites. 

City Year Data 

Data from City Year included years of partnership with City Year, number of corps members per school, school math and 
ELA focus list enrollment and participation, 2 and results from City Year's WSWC Model Configurations Survey. 

Years of City Year partnership. We used these data first to determine, in consultation with City Year, schools' eligibil- 
ity for inclusion in the outcomes analyses. We also used these data to categorize CY schools based on whether they had 
begun their partnership with CY before 201 1-12 (the year when City Year's WSWC model was brought to scale) or after 
(i.e., between 2011-12 and 2013-14). 

Corps member to student ratio. We used these City Year data to create a variable indicating the number of corps mem- 
bers for every 100 students enrolled at a school in grades 3-9 in which City Year corps members serve. 

Percent of students included on City Year ELA or math focus lists, by school, by year. These data captured the number 
of students in grades 3-9 at each school who were included on City Year's ELA and/or math focus lists. We used focus list 
enrollment data, along with state school-level enrollment data, to calculate the percent of students at each school who 
were included on ELA and/or math focus lists. We also determined the median focus list enrollment across all schools for 
the two focus lists, which was 10 percent of the students for ELA and 8 percent of the students for math. 

WSWC Model Configurations Survey (2012-13 and 2013-14). Over a two-year period, City Year program managers, 
who lead teams of AmeriCorps members in delivering the WSWC model in schools, completed an end-of-school year sur- 
vey measuring fidelity of implementation. This survey, which was designed by the City Year Program Design and School 
Relations departments as part of an initiative and larger survey, articulated and measured ways that the WSWC model 
was being implemented. We used these data to construct several indices indicating fidelity of implementation of the 
WSWC model and of specific aspects of the model, including the math and ELA components. The "Overall Implementation 
Index" was constructed from 39 indicators included in the WSWC Model Configurations Survey. The indicators included 
ELA and math focus list implementation; attendance focus list implementation; behavior and social emotional learning 
focus list implementation; implementation of planning and appreciation activities and events; and implementation of 
data reviews, shared planning time, and partnership development. The ELA and math implementation indices were based 
on six indicators, including focus list selection procedures, progress monitoring, frequency of tutoring activities, length of 


2 

In addition to math and ELA focus list data, we received data on the number of students on City Year behavior and attendance focus lists, although 
these data were not complete within or across years. These missing data limited our ability to include attendance and behavior focus list participation 
in our analyses. 
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tutoring sessions, learning contexts (i.e., tutoring schedule), and research-based tutoring (i.e., whether tutoring strategies 
are based on research-based programs or practices). 

State School-level Data 

PSA downloaded school-level data for all City Year schools and potential comparison group schools from state and district 
websites for 2010-11 through 2013-14. As every state designs and implements its own accountability and reporting sys- 
tems, data availability varies by state and not all available data are comparable across states. Accordingly, we describe the 
data we were able to use across the City Year sites, followed by a description of data limitations, including non-comparabil- 
ity. Readers can find a more comprehensive description of the procedures we followed fordownloading and processing 
state data in Appendix B. 

School-level demographic, ELL, and poverty data. Basic data for the percent of students in a school classified as ELL or 
as economically disadvantaged are available for all states, although some states are more comprehensive in their report- 
ing than others (e.g., while some states report the percent of students receiving free or reduced-price lunch [FRPL] as an 
indicator of school-level poverty, other states, such as California, report more comprehensive indicators of school poverty, 
drawing not only on FRPL data but also on Census and state human services databases). School-level race and ethnicity 
data are not as uniformly accessible at the school level in each state, though PSA collected and aggregated these data 
when available. 3 

Graduation rate data. In recent years, states have begun reporting four-year cohort graduation rate data in addition to 
one-year graduation rates. We report four-year cohort data, also sometimes referred to as the National Governor's Associ- 
ation (NGA) graduation rate formula. We used graduation rate and improvement in graduation rate as both independent 
and outcome variables. Models in which we used graduation rate (or an improvement in graduation rate) as a dependent 
variable did not produce meaningful results. When used as an independent variable in the models we report, graduation 
rate was not significant and did not significantly improve model fit. As such, we report graduation rates in our descriptive 
statistics but do not use these data in our logistic or linear regression models. 

Attendance data. Not all states make attendance data available either on their download sites or on school account- 
ability report cards. The availability and overall inconsistency of attendance data among states and across years reduced 
our sample size to such an extent that we found it difficult to meaningfully interpret the models in which we used it as a 
dependent variable. Similarly, we found that its inclusion as an independent variable in our logistic or linear regression 
models either did not significantly alter our findings or so limited our sample size such that interpretation of the results 
proved difficult. Therefore, we opted to use attendance data only in our descriptive reporting and only for the states and 
years for which data were available. 

State ELA and math assessment scale scores. Nine states make assessment scale scores available-Arkansas, California, 
Florida, Michigan, New Hampshire, New York, Rhode Island, South Carolina, and Wisconsin. As each state administers its 
own set of assessments for math and ELA, however, scale scores are not directly comparable across states. Nonetheless, 
scale scores can be made comparable by conversion into standardized z-scores using state-level yearly grade-level mean 
scale scores and standard deviations for each tested subject. We include states for which we were able to access scale 
scores in our fixed-effects linear regression analyses. 


3 

NCES datasets include school-level race and ethnicity data, which we use in our descriptive analyses. These data can be unreliable, however. There- 
fore, in our regression analyses, we use only data collected and reported by states. 
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Data Limitations 


Percent proficient on state ELA and math assessments. States differ not only on the assessments they give students, 
but also on the standards on which students are assessed, the categories they use to classify students' and schools' pro- 
ficiency, and the cut scores used to assign students to proficiency categories. Unfortunately, because of these issues, we 
were unable to use percent proficient data as an outcome variable in linear regression models. To address this, we used 
proficiency data, with state improvement or growth indices, to create the school improvement indicator variables used 
in our fixed-effects logistic regression analyses. If we had percent proficient data for each year, we could code the binary 
outcome variable indicating improvement as "1" if the school's proficiency rate was higher than that of the prior school 
year, and "0" if data indicated a lower proficiency rate. In the absence of percent proficient data-a particular issue for high 
schools-we referenced school improvement or growth indices to code the improvement variable. 

School improvement or growth indices. As for other accountability measures, each state determines the improvement 
or growth index or set of indices they will report, and states are neither consistent in their calculation of these indices nor 
consistent in the particular set of indices they report across years. For example, Washington changed its accountability 
index twice during the four years for which we collected data. Although Louisiana reports a single school performance 
index consistently across years, the state has made changes to its calculation. Texas, meanwhile, changed its reporting 
system after the 201 1-12 school year to include four school accountability indices (achievement, growth, achievement 
gaps, and college and career readiness). As for percent proficient data, these cross-state and cross-year inconsistencies 
prevented us from using scores on state accountability indices as outcome variables. Instead, as described in the previous 
paragraph, in the absence of school-level proficiency rates, we used state growth indices to code the binary school ELA and 
math improvement outcome variables used in our logistic regression analyses. 

School climate data. We were able to download school climate data from a subset of states and districts, including 
Colorado, Massachusetts, Washington, Jacksonville, Los Angeles, New York, and Chicago, fora limited number of years. 
However, because we could not find consistently comparable cross-state or cross-site school climate items or indices, we 
were unable to use the school climate data in the analyses. 

Data availability. For a variety of reasons, many of which are described above, data were not available in some states in 
some years, with no consistent pattern. However, in 2013-14, many states stopped using: (1) their ELA and math assess- 
ments in anticipation of launching new Common Core-aligned assessments; and/or (2) Algebra I end-of-course assess- 
ments. Both of these state testing changes resulted in significantly reduced Ns for the analyses of ELA and math outcomes 
for 2013-14. 

Propensity Score Matching 

We sought to identify two comparison non-CY schools for each CY school in the year before the school began its partner- 
ship with City Year. To find these matches, we first assembled all state achievement, enrollment, and demographic infor- 
mation, and then isolated the district in which CY schools are located. After examining our initial set of matched schools, 
we found that a school's racial composition appeared to be less related to outcomes than were the percentages of students 
identified as English-language learners (ELL) or economically disadvantaged (ED). These findings were consistent with our 
expectations. That is, in districts with a high percentage of students of color, we would expect to see that other variables, 
such as ED and ELL status, would have a larger impact on student achievement than would students' race/ethnicity. As 
such, in our subsequent analyses, we focused on selecting matches based on the percentages of ELL and ED students 
rather than on schools' racial or ethnic composition, and we also considered factors such as attendance rates when data 
were available. 
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To find matches, we conducted two analyses in Stata: psmatch2 and teffects, using nearest-neighbor matching with 
replacement for math and ELA outcomes separately, and including socio-demographic data as our independent variables. 
We were not entirely satisfied with the final group of matched schools determined by either command, as neither can 
independently prioritize the most critical variables for a "good" match. That is, while the matches on percent ED and the 
outcome variable may be good, the suggested matches may be substantively different in the percent of enrolled students 
classified as ELL. Furthermore, particularly in sites where a large number of schools span multiple school levels, such as 
middle-high school combination schools, the routines cannot accurately determine which grades actually contribute to 
the outcome variable, which varies given the state's tested grades. To address these issues for each CY site, we visually 
inspected the suggested comparison schools and made changes to find the most accurate matches across all three match- 
ing variables. Where necessary, such as for CY schools in Manchester, N.H., and Providence, R.I., we pulled in additional 
schools from similarly sized and resourced districts in the state to find accurate matches. In two sites, Boston and Denver, 
we could not find a single set of comparison schools that matched CY schools on both math and ELA outcomes. Therefore, 
for those sites we pulled two sets of comparison schools. Exhibit 1 presents the matching results by school size and by 
student demographic and attendance characteristics, aggregated to the site level. 

Data Analysis 

We conducted three types of analyses to address City Year's research questions, including descriptive, fixed-effects linear 
regression, and fixed-effects logistic regressions for both ELA and math outcomes. Descriptively, we used all available data 
to describe the site and school contexts in which City Year serves, employing these data to identify schools and sites that 
appeared to be implementing the WSWC model or components with greater fidelity and to isolate sites and schools that 
appeared to be having a greater impact on improving student outcomes. These descriptive findings informed our process 
of constructing our linear and logistic analyses. 


Fixed-effects Logistic Regressions 

Our primary cross-site analyses used fixed-effects logistic regression to investigate the relationship between schools' 

City Year team and focus list characteristics and ELA and math improvement between 2011-12 and 2013-14. Logistic 
regressions use a binary outcome variable, indicating success or failure, or, in this case, for each year, whether a school had 
improved in ELA or math. We constructed separate math and ELA outcome indicators, reflecting whether the school had 
made progress in the subject compared with results from the prior school year. To illustrate, if in 2011-12,28 percent of a 
school's students had scored proficient or advanced on the state ELA assessment, whereas in 2010-1 1 only 24 percent had 
scored proficient or advanced, we coded the ELA indicator as "1" for that school in the 201 1-12 school year, capturing the 
improvement in the school's ELA outcomes compared to the previous school year's. We then made a similar comparison 
on math outcomes between 2010-1 1 and 2011-12 to determine whether the school code for math would be a "0," indi- 
cating no improvement or decline, or a "1 ." A description of the method we used to create the binary outcome variables 
is included in Appendix C. We repeated this in both subjects for sets of CY schools and non-CY comparison schools in each 
subject and school year, ensuring that CY schools and their non-CY comparisons received an outcome indicator only for the 
years in which they were partnered with City Year. Thus, a school that began partnering with City Year in 2013-14 would 
not have a recorded outcome variable for 2012-13, for example. 

In all our models, we included site-level, fixed-effects to control for the various state and district contexts in which City Year 
and comparison schools operate. 4 


4 We also included school-level percent ELL and poverty measures as controls in all our logistic and linear models to capture any additional variation 
in these school characteristics not addressed by site fixed-effects or matching. 
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Exhibit 1 

Characteristics of students attending City Year schools and their matched comparisons, 

for SY 201 2 - SY 201 4, by City Year site 


Site name 

Average number of 
students per school 

Percent of students who are 
economically 
disadvantaged 

Percent of students who are 
Limited English Proficient 
(LEP) 

Average student attendance 
rate 

CY schools 
(N=1 49) 

Non-CY 

schools 

(N=465) 

CY schools 
(N=1 52) 

Non-CY 

schools 

(N=486) 

CY schools 
(N=1 50) 

Non-CY 

schools 

(N=478) 

CY schools 
(N=1 08) 

Non-CY 
schools 
(N =358) 

Baton Rouge 

612 

631 

91% 

94% 

3% 

3% 

94% 

91% 

Boston 

543 

429 

84% 

78% 

35% 

33% 

93% 

92% 

Chicago 

637 

872 

98% 

95% 

2% 

6% 

80% 

83% 

Cleveland 

585 

604 

100% 

100% 

33% 

10% 

87% 

88% 

Columbia 

418 

493 

99% 

97% 

0% 

0% 

92% 

94% 

Denver 

937 

749 

84% 

87% 

11% 

26% 

86% 

91% 

Detroit 

370 

420 

80% 

78% 

1% 

16% 

79% 

77% 

Jacksonville* 

664 

1,136 

78% 

71% 

1% 

8% 

99% 

98% 

Little Rock 

473 

630 

90% 

89% 

13% 

9% 



Los Angeles 

741 

1,030 

85% 

88% 

36% 

34% 



Miami 

1,297 

1,556 

94% 

89% 

11% 

14% 

91% 

93% 

Milwaukee 

311 

419 

93% 

90% 

23% 

10% 

86% 

89% 

New Hampshire 

470 

459 

78% 

46% 

0% 

0% 

95% 

95% 

New Orleans 



89% 

90% 

3% 

0% 

76% 

92% 

New York 

745 

764 

92% 

91% 

17% 

19% 

90% 

91% 

Orlando* 

1,451 

1,252 

94% 

88% 

16% 

11% 

93% 

93% 

Rhode Island 

732 

630 

88% 

91% 

0% 

0% 

90% 

93% 

Sacramento* 

386 

491 

100% 

98% 

26% 

34% 



San Antonio* 

918 

857 

88% 

86% 

13% 

16% 

95% 

95% 

San Jose 

464 

376 

87% 

81% 

51% 

45% 



Seattle 

607 

446 

75% 

78% 

16% 

17% 



Washington, D.C. 

165 

177 

100% 

100% 

0% 

11% 



ALL (Mean) 

684 

624 

93% 

86% 

17% 

19% 

92% 

93% 


Exhibit reads: In Baton Rouge, the average number of students in a CY partner school was 612, compared to 631 students in a non-CY school. 
Source: State data, 2012. Sites denoted with an asterisk (*) show data from 2013. 
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Fixed-effects Linear Regressions 


As discussed above, we were able to download school mean ELA and math scale scores for schools in nine states. We 
calculated standardized z-scores for these data and then computed year-by-year change scores for ELA and math outcomes. 
For example, if a school's standardized math score was -0.12 in 201 1-12 and 0.14 in 2012-13, then the school's math 
dependent variable would be 0.26 in 2012-13, or .26 standard deviations, reflecting the school's math improvement from 
2011-1 2. While scale score data were available only for a subset of states and years, we opted to conduct these additional 
analyses because the continuous outcome variable can be more sensitive to small shifts in schools' performance than a di- 
chotomous indicator based on percent proficient metrics. As with our logistic regression analyses, we included a site-level 
fixed effects variable to control for variation across sites. 

Study Limitations 

We made every attempt to locate and equate state data across sites and years. However, given changing state accountabil- 
ity systems between 201 1-12 and 2013-14 and the inherent incomparability of state proficiency data, we were somewhat 
limited in the variables we could use and in the analyses we could conduct. State achievement data are comparable across 
states in limited circumstances and socio-demographic data are often limited. Moreover, states' changing accountability 
systems complicate cross-year and longitudinal analyses. (These issues also impacted the size of our sample in some 
years. 5 ) In addition, the changing set of City Year schools in each school year hindered our ability to conduct longitudinal 
analyses; we could not find an overlapping set of City Year schools with consistent state data sufficiently large enough to 
conduct a three-year analysis. 

In addition, because these data were school- rather than student-level data, we were unable to isolate the effects of the 
model on those students (whom City Year identifies as focus list students) receiving the most intensive academic interven- 
tions. Nevertheless, because the WSWC model is, in fact, a whole-school intervention, measuring City Year's impact on 
school-level performance is consistent with City Year's theory of change. 

Finally, we found that all our City Year variables were highly correlated. To avoid problems with multicollinearity, we do not 
include these variables together in models. 


5 We also opted not to include Philadelphia or Columbus in our analyses because of ongoing investigations in both districts regarding assessment 
irregularities. 
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Characteristics of Partner Schools in City Year Sites 


City Year schools varied by school level and by size. As shown in Exhibit 2, 44 percent of the partner schools were elemen- 
tary schools, 24 percent were middle schools, 25 percent were high schools, and 7 percent were other (i.e., primarily K-8 or 
6-1 2 schools). The schools also varied in size, although most were medium (45 percent) or large (42 percent). Most of the 
high schools partnering with City Year were large (77 percent), with 600 or more students. However, 2 percent of the high 
school partners were small, with fewer than 300 students. In addition, the majority of elementary school partners were 
medium in size, with 300 to 599 students. 


Exhibit 2 

City Year school size, by school level 
SY 2012 -SY 2014 


■ Small (Less than 300 students) ■ Medium (300 to 599 students) ■ Large (600 or more students) 


100 
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More than 93 percent of the schools partnered with City Year are high-poverty (i.e., serve populations in which 75 percent 
or more students are classified as ED). In the majority of City Year sites, 100 percent of City Year schools are high poverty. 
Across all sites, New Hampshire and Seattle serve the lowest percent of high-poverty schools, 67 percent and 78 percent of 
schools, respectively (Exhibit 3). 



Corps Member to Student Ratio 

Across the three school years, City Year sites' partner schools had a median of approximately three corps members for 
every 100 students enrolled in focus grades (grades 3-9). However, the average ratio of corps members to students 
enrolled in focus grades trended downward between 2011-12 and 2013-14, from an average of four corps members to an 
average of three corps members for every 100 students. This change was most likely driven not by the number of corps 
members working at a school but by student enrollment increases in focus grades at some CY high schools. Between 
2011-12 and 201 3-1 4, the corps member to student ratio at CY schools increased in 8 sites, decreased in 10 sites, and 
remained virtually unchanged in 4 sites. 
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Exhibit 4 

Ratio of Corps members (CM) to students in focus list grades in City Year 
partner schools, in 201 1 -1 2, 201 2-1 3, and 201 3-1 4, by City Year site 


2011-12 


Median, all years: 3.3 


IT 


2012-13 


2013-14 




Percent of Students Included on City Year ELA or Math Focus Lists 

On average, the percent of all students enrolled in CY schools' ELA or math focus lists was approximately 12 percent for ELA 
and 8 percent for math. These percentages varied over time and across sites, however. As indicated in Exhibits 5 and 6, 
the median percent of students included on ELA focus lists was 10 while the median percent for math was 8. Although two 
sites (Miami and Baton Rouge) saw notable increases between 2011-12 and 2013-14 in the percent of students served, 
most saw these percentages decline. Of particular note were the large downward trends for San Antonio, Denver, Detroit, 
and Los Angeles for both ELA and math focus lists. 
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Exhibit5 

Percent of City Year schools' students included on 
City Year ELA focus lists in 201 1 -1 2, 201 2-1 3, and 201 3-1 4 



Exhibit 6 

Percent of City Year schools' students included on 
City Year math focus lists in 201 1 -1 2, 201 2-1 3, and 201 3-1 4 


■ 2011-12 ■ 2012-13 ■ 2013-14 


30 


" ■ I Ll N,# I 
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Implementation Index Scores 

The extent to which City Year sites were implementing the WSWC model-as measured by City Year's WSWC Model 
Configurations Survey-varied both across and within sites. Washington, D.C. had the widest range of school-level 
implementation scores and its overall average (62 percent) was relatively low compared with averages from other sites 
(Exhibit 7). Chicago, conversely, had the highest average implementation score (82 percent) and its individual school- 
level implementation scores were tightly clustered around the average, suggesting that there was good site-level 
oversight of model implementation. While the sites varied significantly in terms of the number of partner schools with 
which they worked, this did not appear to correlate with fidelity to model implementation. Indeed, some of the smallest 
and largest sites had both low and high implementation scores. 


Exhibit 7 

Percent of total points earned on implementation index OVERALL, 
by City Year site and partner school 

100 , 



20 


A 



Average Site-Level FULL 
Implementation Index Score 

City Year partner school 

i I i i y 


High Implementing Sites 



We also found variation across and within sites on math (Exhibit 8) and ELA (Exhibit 9) implementation, with no apparent 
implementation patterns related to site-level size. Again, site-level implementation scores varied for both small and large 
sites, although Chicago continued to post high and relatively tightly clustered implementation scores for both ELA and 
math implementation (Exhibit 8). The range in average implementation scores for ELA was from 45 percent (Little Rock) 
to 89 percent (Sacramento) (i.e., of total index points earned). The range was wider for math than for ELA, however, with 
a low of 23 percent (San Jose) to a high of 89 percent (New Orleans). The average implementation score across all sites 
also varied by subject area, with an overall average of 72 percent for ELA implementation versus an overall average of 62 
percent for math implementation. 


16 


Policy Studies Associates, Inc. 



Exhibit8 

Percent of total points earned on MATH implementation index, 
by City Year site and school 



Exhibit 9 

Percent of total points earned on ELA implementation index, 
by City Year site and school 
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Overall City Year Impact on School Performance 


CY schools were more likely to show improvement on state assessments than schools that were similar in terms of their de- 
mographic and performance characteristics but that had not partnered with City Year (non-CY schools). These differences 
were found for performance on state ELA assessments in 2011-12, 2012-13, and 2013-14, and for state math assessments 
in 2011-12 and 2012-13 (Exhibit 10). 


Exhibit 10 

Likelihood that City Year schools improved on 
ELA and Math assessments in 201 2, 201 3, and 201 4 

Likelihood of improvement over prior year 
3 

2.5 

2 

1.5 

1 

0.5 

0 



2011-12 2012-13 2013-14 

qE LA -Math 


* 


Indicates significance, p<=0.05 


ELA Outcomes 

CY schools were approximately two times more likely to improve on state ELA assessments in 2011-12, 2012-13, 
and 2013-14, compared with non-CY schools that shared similar demographic and performance characteristics. For 

2011-12: /?= 1.841 . p=0.018. A/ = 5 1 1 : for 2012-13: /?= 1.97. p<=0.001 ,/V=576: and for 2013-14: /?=1 .89 . p=0.007. 
A/=430. We found particularly distinct school-level effects on CY schools' ELA performance in 2012-13, the yearforwhich 
we had the most complete data across sites. Elementary schools that partnered with City Year in 2012-13 were 3.5 times 
more likely to improve on state ELA assessments than their non-CY comparison schools, while high schools that partnered 
with City Year were three times more likely to improve on their state's measure of high school ELA performance than their 


18 Policy Studies Associates, Inc. 



non-CY comparison schools (elementary school: /7= 3.54. p=0.003, N= 205; high school: /?= 3.00, p=0.022, N= 121). 

The impact of City Year partnership on middle schools' likeliness to improve on state ELA assessments was marginally 
significantin 2012-13: (B= 2.16. p<=0.108, /V=134). 6 

Math Outcomes 

CY schools were significantly more likely to improve on state math assessments in 2011-12 and 2012-13, com- 
pared with non-CY schools that shared similar demographic and performance characteristics. For the 2011-12 math 
assessments, we found that CY schools were 1.7 times more likely to improve than non-CY comparison schools (/?= 1 .73. 
p=0.04, N= 458). Similarly, in 2012-13, CY schools were almost three times more likely to improve on state math assess- 
ments than non-CY comparison schools (/?= 2.86, p<=0.001, /V=534). Finally, for 2013-14, although there was a positive 
association in math, it was not statistically significant (/?=1. 15, p=0.565, A/=386). 

As was the case for EFA, we found distinct school-level effects on math improvement for CY schools in 2012-13. We found 
a marginally significant effect of City Year partnership for elementary schools (/?= 1 .81 . p=0.1 1 , /V=1 94), and notable 
significant effects for middle and high schools. That is, for 2012-13, middle schools that partnered with City Year were 8.8 
times more likely to improve their math performance on the state assessment than middle schools that did not partner 
with City Year. In addition, we found that CY high schools were four times more likely to improve on their school math 
outcome than high schools that shared similar demographic and performance characteristics but that did not partner with 
CY (middle schools: B=8.83. p<=0.001, A/=127; high schools: B=AM. p=0.01 1, /V=104). 


Exhibit 11 

Differences between CYand non-CY schools in predicted likelihood 
of improving on state assessments in 201 2-1 3, by City Year site 

100 


■ Difference in Likelihood between CY and non-CY Schools for ELA 

■ Difference in Likelihood between CY and non-CY Schools for MATH 


60 


40 



6 Interestingly, however, we did find a significantly positive effect of partnership with City Year on middle schools' ELA outcomes in 2013 in our 
z-score analyses, discussed below. 
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School Performance by City Year Site 


On average, CY schools in every City Year site appear to be more likely to improve on ELA and math assessments 
between 2011-12 and 2012-13-the year for which complete data were available for all City Yearsites-than their 
non-CY comparison schools. Indeed, while the likelihood of showing improved performance varied by site, all partner 
schools at City Year sites were more likely to improve on ELA and math performance than their non-CY comparison schools 
(Exhibit 11). We include these data not to encourage site-by-site comparisons, but to demonstrate that no one CY site is 
driving the results and thatthe CY effect holds, and is consistent, across sites. 


Impacts of City Year Model Implementation on Partner School 
Performance 

To better understand the differences in performance among CY schools, we investigated the possible impacts of varia- 
tion in City Year model implementation on schools' performance on state assessments in ELA and math. Specifically, we 
looked at characteristics of model implementation that appeared to vary across sites and schools, including the ratio of 
corps members to students, the percent of students included on focus lists, and whether schools began their partnerships 
with City Year in 201 1-12 or later. Overall, we found that CY schools having a higher ratio of corps members to students, 
enlisting a larger percentage of students on focus lists, and starting their partnership with City Year in 2011-12 or later 
(i.e., when the WSWC model was fully developed) were more likely to show improvement on state assessments in 2012- 
13, compared with (1) partner schools that did not implement characteristics of the model at a high level of fidelity or (2) 
non-CY comparison schools (Exhibit 12). The following section describes the specific impacts of model implementation on 
partner school performance. 


Exhibit 12 


Likelihood that City Year schools improved on ELA and Math assessments, 
by characteristics of City Year model implementation 

Likelihood of improvement over prior year, 2012-13 school year 



CM to student ratio 
■ Began partnership 2011-12 or later 


■ Focus list enrollment 


* Indicates significance. 


p<=0.05 
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Corps Member to Student Ratio 


CY schools with a higher ratio of corps members to students had a higher likelihood of seeing improved ELA 
performance on state assessments in 2012-13. The effects were positive but not significant in 2011-12 and 201 3-14. 
Specifically, CY schools with more than three corps members for every 100 students were 2.5 times more likely to achieve 
improved ELA performance on state assessments, compared with CY schools with fewer than three corps members for 
every 100 students, and compared with non-CY schools (/7= 2.52. p=0.04, A/=563). We saw no differential school-level 
effects (i.e., by elementary, middle, and high schools) on ELA performance among CY schools with more than three corps 
members for every 100 students. 

Similarly, CY schools with a higher ratio of corps members to students had a higher likelihood of seeing improved 
math performance on state assessments in 2012-13. CY schools with more than three corps members for every 
100 students were 3.8 times more likely to improve on their state's math assessment in 2012-13 (/?= 378, p<=0.001, 

N= 527). In addition, middle schools with more than three corps members per 100 students were 8.5 times more likely to 
see improved math performance on state assessments in 2012-13(^= 12.51, p< =0.001 , A/=127). While there was also 
a positive association at the elementary level between the corps member to student ratio and schools' math performance, 
it was only marginally significant for 2012-13 (/?=Z5, p=0.09, N= 118). We saw no differential effects by high school 
level on CY schools' math performance. Finally, as was the case for the ELA analyses, we observed no significant effects on 
math performance across other school years. 


Focus List Enrollment and Service Dosage 

ELA Outcomes 

CY schools whose ELA focus lists included more than 1 0 percent of the students were significantly more 
likely to show improvement on the state ELA assessment in both 2012-13 and 2013-14. 78 In 2012-1 3, CY schools 
whose ELA focus lists included more than 10 percent of the school's total student enrollment were 2.1 times more likely 
to improve on the state ELA assessment, compared with CY schools whose ELA focus lists had fewer than 10 percent of 
the school's total student enrollment, and compared with non-CY schools (/?= 2.11. p=0.007, N= 573). In 2013-14, 
these CY schools whose ELA focus lists were above the median were 1.7 times more likely to improve on their state's ELA 
assessment (/?=17, p=0.036, N= 436). When we examined the association between ELA focus list enrollment and ELA 
outcomes at the school level, we found significant effects in 2012-13 for elementary and high schools, which were 3.3 
and 4.1 times, respectively, more likely to improve on their state ELA assessment (elementary schools: /?= 3.29, p=0.014, 
N= 205; high schools: B= 4.12. p=0.04, N= 134). 


7 The relationship was not significant in 2011-12. 

O 

As described earlier, we calculated the median percent of schools' total enrollment that was included on City Year sites' ELA focus lists. On av- 
erage, across the 24 City Year sites, the median percent of schools' total enrollment included on City Year ELA focus lists was approximately 10 percent. 
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Math Outcomes 


CY schools whose math focus lists included more than 8 percent of the students were significantly more 
likely to show improvement on the state math assessment in both 2011-12 and 2012-13. 9 ' 10 That is, CY schools 
whose math focus lists included more than 8 percent of their students were 2.5 times more likely to improve on their math 
assessment in 2011-12, and 2.3 times more likely to improve in 2012-13 (201 1-12: /?= 2.52, p=0.009, N= 446; 2012- 
13: /7= 2.26. p=0.008, N= 497). In addition, we found differences at the middle school level. That is, CY middle schools 
whose math focus lists included more than 8 percent of students were 4.1 times more likely to improve their performance 
on state math assessments in 201 1-12 and eight times more likely to improve their performance in 2012-13 (2011-12: 
/?= 4.09. p=0.03, A/=92; 2012-13: /?= 7.99. p< =0.001, A/=1 20). No differences were found, however, at the elementary 
and high school levels. 


City Year Partnerships 

We also examined the impact of the length of a school's partnership with City Year on both math and ELA outcomes. 
Specifically, we identified the schools that partnered with City Year before 201 1-12 (the year when the WSWC model was 
implemented at scale), and those that began their partnership with City Year in 2011-12 or after. 

Schools that partnered with City Year in 2011-12 or after were more likely to show improved performance on their 
state's math assessment in 2012-13. Indeed, we found that schools partnering with City Year in 201 1 -1 2 or after were 
three times more likely to improve on their state's math outcome compared with schools that began their partnership 
with City Year before 2011-12 (/?= 3.12. p=0.04,A/= 1 64). For improvement in ELA outcomes, we found a similar associ- 
ation for schools that began their City Year partnership in 201 2-13 (i.e., not 2011-1 2). Schools that began their City Year 
partnership in 2012-13 were approximately three times more likely than other CY schools and comparison non-CY schools 
to improve in ELA outcomes (/?= 2.84. p=0.02, N= 577, [2012-1 3 data]). 


Overall CY Model Implementation 

We also examined the relationship between schools' ELA and math improvement and site-level implementation 
of City Year's WSWC model (based on the results of City Year's 2013-14 WSWC Model Configurations Survey). 

Schools in high-implementing CY sites were 1.8 times more likely to improve on their state's school-level 
ELA outcomes, compared with schools in low-implementing CY sites and non-CY comparison schools 1/3= 1.84, 
p<=0.05, N=432). Among the high- and low-implementing CY schools, we found an association between high site-lev- 
el implementation and improved school performance on the state ELA assessment (/?= 2.1 1. p=0.054, N= 139; [2013-14 
data]). We did not, however, find a significant effect on math outcomes using the overall implementation index. Nonethe- 
less, when we limited the implementation index to just the measures of math implementation, we found that schools in 
CY sites with high math implementation scores were 2.5 times more likely to improve on their state's math assessment in 
2013-14, compared with CY schools in sites with low math implementation (/?= 2.5, p=0.024, A/=1 25). We did not find 
a similar relationship, however, between schools in CY sites with high ELA implementation and performance on state ELA 
assessments. 


0 

As described earlier, we calculated the median percent of schools' total enrollment that was included on City Year sites' math focus lists. On 
average, across the 25 City Year sites, the median percent of schools' total enrollment included on City Year math focus lists was approximately 8 
percent-slightly lowerthan was the median percentfor ELA. 

10 

The relationship was notsignificant in 2013-14. 
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Other City Year Impacts: Scale Scores 


For the subset of sites and school years (i.e., 201 1 -1 2 to 2012-1 3) for which we had the most scale score data, we analyzed 
schools' change in math and ELA scores. 11 ' 12 We note, however, that the proportion of the variance that is explained by the 
models (R 2 ) is low, suggesting that the results should be interpreted with caution. In addition, we found that including 
more than one City Year implementation variable in a model introduced multicollinearity. That is, although all the City 
Year measures (e.g., percent of total enrollment on the math or ELA focus list, implementation rank, corps memberto 
student ratio) capture slightly different phenomena, they are, nonetheless, highly correlated. 

In all the models using scale scores as the dependent variable, we included variables that accounted for each of the 
following: 

Site-level fixed effects — a variable for each site that captures unknown differences among the states and districts 
in which CY schools operate 

Percent ELL — percent of the school's enrollment classified as English Language Learners (ELL) 

Percent high poverty — percent of a school's enrollment identified as high poverty 13 


Schools partnered with City Year gained approximately one month of additional ELA learning, as measured on the 
state ELA assessment, or 0.07 standard deviations (/3= 0.074. p=0.02, n=313), compared with non-CY schools. At 

the school level, middle schools that partnered with City Year improved their performance on ELA assessments by 0.12 
standard deviations (/?= 0.1 15, p=0.013, n=107). This translates to approximately one month of learning gains as 
measured by the state assessment. 14 There was no significant association at the elementary level (/?= 0.107. p=0.314, 
/?=123) or high school level (/?= 0.384 . p=0.121,/i=70). The lack of significance at the high school level, however, may 
be related to the relatively low number of schools for which there were adequate data available. 

Schools that partnered with City Year achieved approximately one month of additional math learning, or 0.08 
standard deviations, on their state's math assessment (/i= 0.080. p=0.04, n=258), compared with non-CY schools. 

As with the relationship between school level and performance on ELA assessments, we found a significantly positive 
effect on math associated with a middle school's CY partnership. Middle schools partnered with CY gained approximately 
0.09 standard deviations on 2012-13 state math assessments, or approximately one month of additional math learning 
f/?=0.088. p=0.025. n=1 07). 


11 

We standardized the scale scores across the City Year sites included in the analyses, converting CY partner schools' and comparison non-CY 
schools' scale scores into z-scores. 

12 

Only a few states release scale scores, including Arkansas, California, Florida, Michigan, New Hampshire, New York, Rhode Island, South Carolina, 
and Wisconsin. Accordingly, these analyses include the following 13 of City Year's 24 sites: Columbia, Detroit, Jacksonville, Little Rock, Los Angeles, 
Miami, Milwaukee, New Hampshire, New York, Orlando, Rhode Island, Sacramento, and San Jose. These findings are not applicable to the entire 
network of City Year sites. That is, these analyses are based only on a subset of City Year sites and partner schools for which scale score data were 
available. 

13 

Although we used ELL and ED variables to construct our comparison group, we included these variables to capture any residual effects associated 
with those student characteristics. 

14 

We interpreted these results by assuming that students generally improve by approximately one standard deviation over the course of a 
10-month school year. One-tenth of a standard deviation, therefore, translates into approximately one month of improvement. 
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CY schools whose ELA focus lists included more than 1 0 percent of their students achieved approximately one 
month of ELA learning gains, as measured on state ELA assessments, compared with all other CY schools and with 
non-CY comparison schools, although this impact was only marginally significant. Specifically, CY schools whose 
ELA focus lists included more than 10 percent of their students saw an increase of 0.07 standard deviations on the state 
ELA assessment (/?= 0.066. p=0.074, n=312). In addition, CY middle schools whose focus lists included more than 10 
percent of their students saw an increase of approximately one month of ELA learning (or 0.1 of a standard deviation) on 
the state ELA assessment (/?= 0.096, p=0.021,n=106), compared with CY schools whose ELA focus lists included fewer 
than 10 percent of their students and compared with non-CY schools. 

CY schools whose math focus lists included more than the median percent (8 percent) of students achieved approx- 
imately 1.5 months of math learning gains (or 0. 16 standard deviations) on state math assessments 1)3= 0.158. 
p<=0.005, n=238), compared with CY schools whose math focus lists included fewer than the median percent 
of students and with non-CY comparison schools. Among CY schools, there was a marginally significant association 
between schools whose focus lists included more than 8 percent of their students and performance on state math assess- 
ments (/?=0J91, p< =0.055, n= 72). In addition, the impact of math focus list enrollment was significant for middle 
schools, where a higher than median percent of students included on math focus lists was associated with a 0.21 standard 
deviation increase in schools' performance on state math assessments, approximately two months of additional learning 
(J3= 0.206, p=0.002, /?=1 03), compared with state math assessment performance results from other CY schools and from 
non-CY comparison schools. This relationship is not significant, however, for either elementary school or high school. 

A higher ratio of corps members to students in a CY school had a significantly positive effect on school-level ELA 
and math assessment outcomes. 

■ For ELA. a higher ratio of corps members to students (i.e.. more than three corps members for every 100 students) was 

associated with an additional month of ELA learning gains 1/3= 0.1 1 1 . p=0.012, n=300). In addition, middle schools 
with a higher ratio of corps members to students scored significantly higher on state ELA assessments, gaining just 
over two months of ELA learning, or 0.23 standard deviations higher(/?= 0.226 p=0.025, n=1 06). 


■ For math, a higher ratio of corps members to students (i.e., more than three corps members for every 100 students) in 

a CY school was associated with an additional month and a half of math learning gains on state-level math assess- 

ments. More than three corps members for every 100 students in a school was associated with an increase of 0.17 
standard deviations on a CY school's state math assessment scores (/?= 0.172, p<=0.01, n=246). Middle schools, 
in particular, benefitted from a higher ratio of corps members to students, which was associated with an increase of 
0.3 standard deviations-or three months of math learning gains-on a school's state math assessment (/?= 0. 303, 
p=0.01,n=106). 


Schools in their first year of partnership with CityYearin 2012-13 performed significantly better on their states' 
math and ELA assessments than CY schools with longer City Year partnerships or non-CY comparison schools. Fo r 

math assessments, the "first CY year" effect was associated with approximately an additional month and a half of math 
instruction, as measured by state math assessments, and a month of additional ELA learning (or a 0.16 standard deviation) 
(Math: /?=0.158. p=0.018, n=258; ELA: 1= 0.113, p=0.027, n=313). 

We also found no significant relationship between school performance on state ELA and math assessments and the imple- 
mentation indices derived from either the 2012-13 or 2013-14 WSWC Model Configurations Survey. 
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APPENDIX A 

High Performing City Year Partner Schools 


Impacts of City Year's Whole School Whole Child Model on Partner Schools' Performance 25 


High-Performing CY Partner Schools 


To identify the high-performing schools, we looked first at our data to identify the schools that made the most math and 
ELA progress between the 2012-13 and 2013-14 school years. We then downloaded state report cards for 2013-14 for 
every CY school to access additional growth metrics and comparative data not captured in state dataset downloads. For 
example, New York City's School Quality Snapshot reports include information on parent satisfaction, progress in closing 
achievement gaps, and students' improvement on the state assessment. The reports also allow for comparison between 
schools as well. We used the schools we initially identified as a starting point for our analysis of school report cards for 
all CY schools. As shown in Exhibit 1 5, the 20 schools we selected are all schools in which all students made significant 
gains in ELA and math in comparison to other schools (i.e., they were identified as "high growth" schools by their state or 
district). These schools also made progress on indicators such as closing the achievement gap and growth for students 
identified as ELLor students with disabilities. 
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a Small schools=<300 students; Medium schools=300-599 students; Large schools=600+ students 


APPENDIX B 

Data Sources and Regression Models 
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Data Sources 


NCES Data 

NCES data were available for both the 2011-12 and 2012-13 school years. However, after merging these files with 
state-level data, we discovered discrepancies between NCES-reported and state-reported data. Most importantly, we found 
that NCES data underreported free and reduced-price lunch enrollment as compared with state data. Consequently, we 
elected to use state data for our propensity matching and in our subsequent analyses. Also, NCES data do not include the 
percent of students classified as ELL. English-proficiency is a state-level classification and differences in state measures of 
ELLs may limit its collection atthe federal level. 


State Data 

Across all sites, we followed the same procedure for collapsing grade-level data into school-level achievement metrics and, 
where scale scores were available, for calculating z-scores. For each state and year, we collapsed each school's grade-level 
cases into a single school-level case, creating and using a weighting variable to ensure that enrollment in each grade 
level was proportionally represented in the final school-level case. This ensured that if, for example, a school enrolled 90 
third-graders and 30 fourth-graders, the third-grade data would be given three times the weight applied to the fourth- 
grade data. PSA calculated z-scores for each year and subject using state ELA and math assessment scale scores and grade 
and year specific state-level standard deviations. 

Louisiana (Baton Rouge and New Orleans) 

Between 2011-12 and 2013-14, Louisiana changed their accountability rankings and measures. To construct the 
school-level measure of student performance for each year, we merged four types of datasets released annually by the 
Louisiana State Department of Education: school performance scores; iLEAP and LEAP assessments; end-of-course assess- 
ments; and school enrollment data. Most of these data for the last two years are available in Excel format on the state's 
website, while some older data are only available in PDF files. Some data, such as attendance rates, were not available. 

In New Orleans, the structure of the Recovery School District and charter networks and the regularity with which schools 
open and close each year complicated this task. States typically do not release data for schools during their first year of 
operation, and the changes the state made to its accountability system over the 2012-13 and 2013-14 school years further 
delayed the release of accountability scores for some schools in their first few years of operation. For schools in both East 
Baton Rouge and New Orleans, where data were not available in a downloadable form, PSA consulted individual schools' 
report cards and manually entered missing data. 

While we found suitable matches for all schools in Louisiana, the regular opening and closing of schools, particularly in 
New Orleans, required careful attention to the year in which a school opened (or closed) and the availability of outcome 
data for the potential matches over the course of the study years. 

Massachusetts (Boston) 

Massachusetts releases school-level data for race/ethnicity, school performance indicators (e.g., attendance rate, retention, 
dropout rate), and for special student populations such as percent ELL. In addition, the state releases grade-level assess- 
ment data, which we aggregated up to the school level in each year. 
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In every City Year site, we attempted to match schools based on school outcome data for the year prior to the start of a 
school's City Year partnership. In Boston, this required attention not only to the opening, closing, and reconfiguration of 
schools, but also to whether City Year began working with the potential match school in a subsequent year. Because City 
Year serves a large proportion of the high-poverty, low-achieving schools in Boston, finding matches was challenging. 
Nonetheless, by weighting comparison sites, we were able to find suitable matches for all CY schools. In addition, as 
mentioned in the section describing our propensity matching process, we could not find a single set of schools in Boston 
that matched CY partner schools on both math and ELA outcomes. As such, we pulled a separate set of comparison schools 
for math and ELA outcomes. 

Illinois (Chicago) 

Chicago Public Schools offers easily downloadable files on its website, including data particularly relevant to City Year's 
work with middle schools and ninth-graders (e.g., high school readiness, school-level ACT EXPLORE average scores, and al- 
gebra assessment results). Matching schools in Chicago was complicated, however, by the districts' recent round of school 
closures and changes in schools' grade-level configurations that affected both CY partner schools and potential compari- 
son schools. Although we were ultimately successful in finding comparison schools, we opted to weight some comparison 
schools to ensure suitable matches in light of the significant percent of the lowest-performing, highest-poverty schools 
City Year serves in Chicago. 

Ohio (Cleveland) 

For the CY partner schools in Cleveland, we merged spreadsheets downloaded from the state's DOE website. Although 
Ohio releases a significant amount of school-level data, the state made some changes to the metrics it uses for school ac- 
countability between the 2011-12 and 201 3-1 4 school years. Although this did not preclude our use of the states' percent 
proficient grade level, it limited our ability to use the state's achievement index in any growth analyses. 

We opted not to use data from Columbus schools in our analyses due to ongoing state investigations into the reliability of 
the district's reported data. 

South Carolina (Columbia) 

South Carolina releases a wealth of school-level data. To prepare the data for analyses, we merged demographic, poverty, 
and achievement index data, and aggregated grade-level percent proficient and scale scores into school-level measures by 
subject and year. Though the state changed its achievement index over the course of the three years of our study, the state 
assessment remained constant and we were able to use these percent proficient data to construct our outcome variables. 

Finally, City Year serves schools in two districts in South Carolina: Lexington 4 and Richland 1. In both sites, the relatively 
small number of schools in the districts required that we weight matches for the high schools and middle schools City Year 
serves. 

Colorado (Denver) 

Constructing the comparison group for schools in Denver presented a challenge. While we did not include the alternative 
schools that CY served for a single year during the 2012-13 school year, the changing configuration of schools in Denver, 
particularly at the high school level, necessitated the weighting of matched high schools. CY serves a large proportion of 
Denver's high-poverty, low-performing high schools, and two of the schools with which CY partnered in the 2012-13 and 
2013-14 school years were new, small schools for which there were few matches on enrollment size. 
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As discussed in the propensity matching section, we could not find a single set of schools that matched CY partner schools 
on both ELA and math outcomes. For this reason, we chose to create two separate groups of comparison schools. 

Michigan (Detroit) 

Michigan's Department of Education posts a large amount of publicly-available data on its website. PSA downloaded com- 
bined files containing all school data for 2010-11, 2012-13, and 2014-15, but no combined file was available for 2011-12. 
To complete the dataset, PSA staff downloaded and merged the files needed for the analysis for that missing year (e.g., 
MEAP, MME, enrollment, student count). We then cleaned and coded these data to ensure thatthe critical variables would 
be consistent with data from other states. 

To prepare the data for school-level matching, we calculated z-scores for the state ELA and math assessment scale scores 
and collapsed each school's grade-level cases into a single school-level case. 

Once we completed the collapsing process for each year, we merged all years into a single dataset to find within district 
matches for each City Year school for the year prior to the start of their partnership with City Year. Two schools included 
in the Detroit site, River Rouge High School and Harper Woods High School, are not part of Detroit Public Schools (DPS). 
After attempting to locate comparable schools with their respective districts, PSA opted to match these schools to schools 
within DPS. Further complicating the matching process, City Year serves a large proportion of the high-needs high schools 
operational in DPS, and, additionally, DPS has closed or consolidated a number of schools in the last few years. Nonethe- 
less, by weighting matched high schools, we were able to find comparable schools in the match year for each CY partner 
school. 

Florida (Jacksonville, Miami, Orlando) 

We downloaded school-level data for Jacksonville, Miami, and Orlando from the state DOE website and then aggregated 
grade-level proficiency and scale score data into single cases for each school, subject, and year. While Florida offers a 
tremendous amount of data about its schools over many years, the state does not release school-level attendance data 
past the 2011-12 school year and attendance data are not available through school report cards. (The state used atten- 
dance data as part of its accountability metrics for the 201 1 -1 2 school year and prior years but changed its accountability 
measures in 2012-13). To construct the comparison group for high schools in Miami, PSA weighted comparison schools' 
data, as City Year Miami serves many of the larger, high-poverty, low-performing high schools in the district. 

Arkansas (Little Rock) 

We used data from two sources to construct our variables for schools in Arkansas. That is, the state Department of Edu- 
cation and the Office for Education Accountability (OEA) at the University of Arkansas offer downloadable files cataloging 
state achievement data, although the OEA's files are often more usable. We merged files from both sources to obtain a 
full dataset for Arkansas schools across all years. A change to the state's unique school ID, however, complicated merging 
files across years. In addition, as was the case in many of the small districts in which City Year operates, because CY serves 
many of the lowest performing schools in the district, we weighted the matched comparison schools. 

California (Los Angeles, Sacramento, San Jose) 

Data were available in multiple files, which we merged using the state school ID (cds_code). California releases assess- 
ment data by grade level, and to create school-level math and ELA measures, we aggregated grade-level proficiency and 
scale score data, by subject and year, using a grade weight, as described above. However, because the state changed 
assessments between the 2012-13 and 2013-14 school years and opted not to release 2013-14 accountability reports for 
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schools, we do not have outcome data for California for 2013-14. While the state did release CAASP results for the 2013- 
14 administration, absent accountability reporting or a translation between the cut scores used for the STAR assessment 
in 2012-13 and 2013-14, PSA was notable to use the CAASP 2013-14 assessment data. Additionally, while California 
releases extensive school-level data, we could not locate school-level attendance data. 

New Hampshire (Manchester) and Rhode Island (Providence) 

New Hampshire and Rhode Island are members of the New England Common Assessment program, though both will 
be transitioning to new assessments in the coming school year. Rhode Island will administer the Smarter Balanced 
assessment while Rhode Island's students will take the PARCC assessment. For years prior to 2014-15, however, the states 
released their data in a similar format on their DOE websites. Rhode Island offers additional school-level data such as 
school-level attendance rates on their state data website (InfoWorks). New Hampshire releases yearly attendance rate data 
at the district level rather than at the school level. 


Wisconsin (Milwaukee) 

Data were easily available for Milwaukee schools on the Wisconsin DOE website. However, Milwaukee has closed or recon- 
figured a number of schools over the last three years, which complicated the matching process, and we opted to weight 
matching schools to ensure comparability with the CY partner schools. 

New York (New York) 

Through the New York City Department of Education's (DOE) and the state DOE's websites, we were able to access 
school-level data. While we primarily relied on the NYC DOE's data for propensity matching, some needed data were only 
available through state databases. Also, NYC releases its data using DBN as the school's unique ID, whereas the state uses 
its own school identification code for its data files. Linking these two sets of data required a translation between the two 
sets of IDs. 

Pennsylvania (Philadelphia) 

We opted not to include schools in Philadelphia in our analyses due to an ongoing investigation into testing irregularities 
at a number of schools in the district. 

Texas (San Antonio) 

Texas releases extensive data about its schools. Unfortunately, for the purposes of this study, the state changed its assess- 
ment in 2012-13. Though the state released the percent proficient metrics for 2012-13, it did not release accountability 
data for the 2012-13 school year. The state does not offer a translation between the proficiency metrics or cut scores for 
two assessments, as did Florida when the state moved to FCAT 2.0 from the original FCAT. As such, our analyses of sites in 
Texas are limited to the 2013-14 school year. 

Washington (Seattle) 

Although the Office of the Superintendent of Public Instruction's (OSPI) website provides a readily accessible system for 
downloading school-level data, the available data do not include attendance data, nor could we find school-level atten- 
dance data on Seattle Public School's website. 
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Regarding outcome measures, we used percent proficient metrics to construct our outcome measure, aggregating 
grade-level data to construct school-level cases. Washington's receipt and subsequent loss of their NCLB waiver between 
2011-12 and 2013-14, however, resulted in near constant changes in their accountability metrics, which ultimately limited 
our ability to use their growth or achievement measures because those measures could not be compared across years. 

Washington, D.C. 

Washington, D.C. releases limited data on its schools. Percent proficient data are available on its website, as are data on 
school graduation rates. To find data on ELLs, we located the number of students taking the ELL-version of the DC CAS for 
each grade and each school in the city. We then summed these data to the school level and used these data with the num- 
ber of students taking the regular DC CAS and the total school enrollment to calculate an approximate percent of students 
enrolled at each school. 

Because a number of CY partner schools began and ended their CY partnership between 2011-12 and 2013-14, we were 
careful to select matches that had not partnered with City Year in previous years and that were unlikely to partner with City 
Year in subsequent years. 
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Regression Models 


Fixed-effects Logistic Regression Results 


Exhibit B-1 

City Year schools' likelihood of improved achievement in ELA and math compared 
with non-City Year schools, by school year 


ELA 

Math 


2010-11 to 
2011-12 

2011-12 to 
2012-13 

2012-13 to 
2013-14 

2010-11 to 
2011-12 

2011-12 to 
2012-13 

2012-13 to 
2013-14 

City Year school 

1.841* 

[0.475] 

1.971** 

[0.406] 

1.888** 

[0.446] 

1.727* 

[0.448] 

2.864*** 

[0.649] 

1.149 

[0.277] 

School-level percent 
poverty 

0.959*** 

[0.007] 

1.017** 

[0.006] 

0.962** 

[0.012] 

0.974*** 

[0.007] 

0.997 

[0.006] 

0.990 

[0.013] 

School-level percent ELL 

1.006 

[0.007] 

1.008 

[0.007] 

1.014 

[0.009] 

0.997 

[0.008] 

1.009 

[0.008] 

1.011 

[0.011] 

Constant 

2.461*** 

[2.576] 

0.158* 

[0.118] 

1.579 

[1.665] 

5.676*** 

[5.6932] 

0.755 

[0.571] 

0.885 

[1.216] 

N 

577 

576 

430 

458 

534 

386 

Pseudo R 2 

0.132 

0.149 

0.125 

0.0922 

0.156 

0.0915 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


Exhibit B-2 

City Year schools' likelihood of improved achievement in ELA and math compared with 
non-City Year schools, from 2010-11 to 2011-12. by grade level 




ELA 


Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school 

1.680 

1.044 

5.544 

1.128 

2.322 

2.241 

[0.684] 

[0.575] 

[5.124] 

[0.479] 

[1.331] 

[1.354] 

School-level percent poverty 

0.994 

[0.020] 

0.993 

[0.041] 

1.000 

[0.053] 

1.004 

[0.021] 

0.919 

[0.044] 

0.976 

[0.059] 

School-level percent ELL 

1.002 

[0.011] 

1.006 

[0.026] 

1.047* 

[0.024] 

1.005 

[0.013] 

1.019 

[0.029] 

0.978 

[0.024] 

Constant 

0.760 

3.643 

3.046 

1.822 

4.431 

24.055 

[1.589] 

[14.517] 

[14.129] 

[3.935] 

[20.281] 

[126.148] 

N 

7 23 

7 07 

77 

7 23 

7 07 

70 

Pseudo R 2 

0.314 

0.263 

0.452 

0.392 

0.393 

0.456 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 
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Exhibit B-3 

City Year schools' likelihood of improved achievement in ELAand math compared with 
non-City Year schools, from 2011-12to 2012-13. by grade level 




ELA 


Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school 

3.535** 

[1.521] 

2.164 

[1.040] 

3.000* 

[1.443] 

1.814* 

[0.679] 

8.834*** 

[4.578] 

4.040** 

[2.571] 

School-level percent poverty 

0.920** 

[0.027] 

1.034 

[0.038] 

0.890* 

[0.048] 

0.940* 

[0.026] 

1.001 

[0.038] 

0.972 

[0.058] 

School-level percent ELL 

0.990 

[0.011] 

1.020 

[0.027] 

1.071* 

[0.029] 

1.006 

[0.012] 

0.958 

[0.033] 

1.030 

[0.028] 

Constant 

3.28** 

[9.039] 

0.022 

[0.069] 

6.243 

[7.973] 

4.370* 

[12.723] 

0.222 

[0.724] 

3.24 

[9.187] 

N 

205 

134 

121 

194 

127 

104 

Pseudo R 2 

0.314 

0.263 

0.452 

0.392 

0.393 

0.456 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


Exhibit B-4 

City Year schools' likelihood of improved achievement in ELAand math compared with 
non-City Year schools, from 2012-13 to 2013-14. by grade level 


ELA 

Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school 

1.187 

1.926 

2.035 

0.575 

1.938 

0.927 

[0.556] 

[0.948] 

[1.202] 

[0.284] 

[0.928] 

[0.481] 

School-level percent poverty 

0.975 

[0.027] 

0.975 

[0.029] 

0.964 

[0.035] 

1.021 

[0.034] 

0.996 

[0.029] 

0.948 

[0.050] 

School-level percent ELL 

1.030 

[0.018] 

1.019 

[0.021] 

0.973 

[0.025] 

1.012 

[0.018] 

1.017 

[0.027] 

1.031 

[0.029] 

Constant 

3.928 

7.099 

9.805 

0.384 

0.995 

37.670 

[10.769] 

[20.514] 

[32.529] 

[1.255] 

[2.762] 

[169.288] 

N 

128 

96 

89 

106 

93 

98 

Pseudo R 2 

0.137 

0.0797 

0.162 

0.106 

0.0698 

0.105 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 
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Exhibit B-5 

City Year schools' likelihood of improved achievement in ELAand math compared with 
non-City Year schools and with City Year schools with a corps member to 
student ratio of fewer than 3:100. by year 


ELA 

Math 


2010 to 

2011-12 to 

201 2-1 3 to 

2010 to 

2011-12 to 

201 2-1 3 to 


2011-12 

2012-13 

2013-14 

2011-12 

2012-13 

2013-14 

City Year school with more 
than 3 corps members for 
every 100 students 

1.242 

1.704* 

1.357 

1.533 

3.590*** 

1.070 

[0.448] 

[0.118] 

[0.419] 

[0.603] 

[1.104] 

[0.338] 

School-level percent poverty 

0.959*** 

[0.007] 

1.016** 

[0.006] 

0.972* 

[0.012] 

0.973*** 

[0.007] 

0.994 

[0.006] 

0.998 

[0.013] 

School-level percent ELL 

1.006 

[0.007] 

1.010 

[0.007] 

1.008 

[0.009] 

0.997 

[0.008] 

1.007 

[0.008] 

1.007 

[0.010] 

Constant 

2.109*** 

0.180* 

4.554 

6.884*** 

0.969 

0.433 

[2.219] 

[0.133] 

[5.973] 

[6.537] 

[0.736] 

[0.603] 

N 

577 

563 

423 

458 

527 

385 

Pseudo R 2 

0.124 

0.147 

0.0986 

0.0868 

0.149 

0.0887 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


Exhibit B-6 

City Year schools' likelihood of improved achievement in ELA and math from 2010-11 to 2011-12 
compared with non-City Year schools and with City Year schools with a corps member to student 

ratio of fewer than 3:100. by grade level 




ELA 


Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school with more 
than 3 corps members for 
every 100 students 

1.898 

0.986 

5.544 

2.202 

1.026 

4.188 

[0.859] 

[0.946] 

[5.124] 

[1.074] 

[1.007] 

[3.904] 

School-level percent poverty 

0.996 

[0.020] 

0.997 

[0.041] 

1.000 

[0.053] 

1.001 

[0.021] 

0.934 

[0.043] 

0.978 

[0.059] 

School-level percent ELL 

1.003 

[0.011] 

1.005 

[0.026] 

1.047* 

[0.024] 

1.006 

[0.013] 

1.012 

[0.028] 

0.975 

[0.024] 

Constant 

0.612 

2.520 

3.046 

2.204 

1,269.046 

19.117 

[1.273] 

[9.970] 

[14.129] 

[4.783] 

[5,597.987] 

[100.652] 

N 

197 

100 

74 

178 

97 

98 

Pseudo R 2 

0.113 

0.177 

0.106 

0.125 

0.185 

0.0315 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


36 


Policy Studies Associates, Inc. 


Exhibit B-7 

City Year schools' likelihood of improved achievement in ELA and math from 201 1-1 2 to 2012-13 
compared with non-City Year schools and with City Year schools with a corps member 
to student ratio of fewer than 3:1 00. by grade level 




ELA 


Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school with more 
than 3 corps members for 
every 100 students 

3.839 

6.744 

5.800 

1.628 

1 2.54** 

13.431 

[1.890] 

[6.532] 

[3.648] 

[0.687] 

[11.59] 

[14.641] 

School-level percent poverty 

0.919** 

1.033 

0.886* 

0.940* 

1.022 

0.977 

[0.027] 

[0.037] 

[0.051] 

[0.026] 

[0.037] 

[0.063] 

School-level percent ELL 

0.986 

1.039 

1.079** 

1.000 

0.975 

1.027 

[0.012] 

[0.029] 

[0.030] 

[0.012] 

[0.031] 

[0.027] 

Constant 

3,405.023** 

0.020 

44,368.116* 

433.741* 

0.053 

12.324 

[10,698.54] 

[0.061] 

[227,003.1] 

[1,287.202] 

[0.169] 

[68.741] 

N 

20 7 

77 8 

727 

7 90 

727 

706 

Pseudo R 2 

0.230 

0.155 

0.169 

0.134 

0.219 

0.125 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


Exhibit B-8 

City Year schools' likelihood of improved achievement in ELA and math from 2012-13 to 2013-14 
compared with non-City Year schools and with City Year schools with a corps member to student 

ratio of fewer than 3:100. by grade level 


ELA 

Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school with more 
than 3 corps members for 
every 100 students 

0.840 

1.007 

1.570 

0.461 

12.5162 

0.495 

[0.402] 

[0.838] 

[1.093] 

[0.237] 

[12.833] 

[0.328] 

School-level percent poverty 

0.979 

[0.027] 

0.980 

[0.029] 

0.978 

[0.037] 

1.020 

[0.033] 

1.001 

[0.029] 

0.957 

[0.049] 

School-level percent ELL 

1.032 

[0.018] 

1.019 

[0.021] 

0.951 

[0.029] 

1.007 

[0.019] 

1.024 

[0.028] 

1.023 

[0.029] 

Constant 

2.799 

5.916 

4.391 

0.540 

0.243 

18.024 

[7.480] 

[17.027] 

[14.924] 

[1.722] 

[0.702] 

[79.569] 

N 

724 

96 

80 

7 02 

93 

95 

Pseudo R 2 

0.121 

0.0665 

0.156 

0.109 

0.0658 

0.0991 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 
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Exhibit B-9 

City Year schools' likelihood of improved achievement in ELAand math compared with non-City 
Year schools and with City Year schools with more than the median percent of students on ELA 

and math focus lists *, by year 



ELA 



Math 


2010 to 

2011-12 to 

2012-13 to 

2010 to 

2011-12 to 

201 2-1 3 to 

2011-12 

2012-13 

2013-14 

2011-12 

2012-13 

2013-14 


More than the median per- 
cent of a school's enrollment 
is on a focus list 1 

1.328 

[0.385] 

2.109** 

[0.589] 

1.681* 

[0.191] 

2.520** 

[0.893] 

2.256** 

[0.690] 

1.126 

[0.365] 

School-level percent poverty 

0.959*** 

[0.007] 

1.017** 

[0.006] 

0.967** 

[0.012] 

0.973*** 

[0.007] 

0.995 

[0.006] 

0.994 

[0.013] 

School-level percent ELL 

1.006 

[0.007] 

1.007 

[0.008] 

1.009 

[0.009] 

1.000 

[0.008] 

1.007 

[0.008] 

1.008 

[0.011] 

Constant 

2.290*** 

[2.553] 

0.166* 

[0.124] 

6.940 

[9.025] 

5.191*** 

[5.167] 

1.021 

[0.776] 

0.616 

[0.845] 

N 

51 7 

573 

436 

446 

497 

375 

Pseudo R 2 

0.125 

0.147 

0.111 

0.0997 

0.121 

0.0937 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


'The median ELA focus list enrollment is 10 percent of the school's total enrollment. In math, the median focus list enrollment is 8 percent of 
schools' total enrollment. 


Exhibit B-10 

City Year schools' likelihood of improved achievement in ELA and math from 2010-11 to 2011-12. 
compared with non-City Year schools and with City Year schools with more than the 
median percent of students on ELA and math focus lists *, by grade level 



ELA 



Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 


More than the median per- 
cent of a school's enrollment 
is on a focus list 1 

1.538 

[0.692] 

0.914 

[0.533] 

4.822 

[7.173] 

2.031 

[1.053] 

4.090** 

[1.942] 


School-level percent poverty 

0.997 

[0.020] 

0.994 

[0.041] 

1.007 

[0.054] 

1.009 

[0.022] 

0.901* 

[0.047] 

0.988 

[0.059] 

School-level percent ELL 

1.002 

[0.011] 

1.006 

[0.026] 

1.047* 

[0.024] 

1.008 

[0.013] 

1.043 

[0.031] 

0.980 

[0.024] 

Constant 

0.567 

[1.180] 

3.548 

[14.134] 

1.611 

[7.467] 

1.018 

[2.246] 

0.303 

[1-26] 

8.538 

[44.681] 

N 

197 

100 

74 

172 

92 

92 

Pseudo R 2 

0.116 

0.174 

0.117 

0.143 

0.247 

0.0324 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 

'The median ELA focus list enrollment is 10 percent of the school's total enrollment. In math, the median focus list enrollment is 8 percent of schools' 
total enrollment. 
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Exhibit B-11 

City Year schools' likelihood of improved achievement in ELAand math from 201 1-1 2 to 2012- 
13, compared with non-City Year schools and with City Year schools with more than the median 
percent of students on ELA and math focus lists, by grade level 


ELA 

Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

More than the median percent 
of a school's enrollment is on a 
focus list 1 

3.296* 

[1.590] 

1.552 

[0.803] 

4.117* 

[1.330] 

0.814 

[0.442] 

7.989*** 

[4.910] 

1.218 

[0.815] 

School-level percent poverty 

0.923** 

[0.027] 

1.038 

[0.037] 

0.896* 

[0.047] 

0.937* 

[0.027] 

0.996 

[0.038] 

0.974 

[0.054] 

School-level percent ELL 

0.991 

[0.011] 

1.015 

[0.026] 

1.068* 

[0.028] 

1.002 

[0.012] 

0.969 

[0.031] 

1.022 

[0.026] 

Constant 

2.324 

[6.660] 

0.018 

[0.055] 

1.315 

[7.654] 

700.421* 

[2,166.671] 

0.350 

[1.137] 

15.064 

[74.335] 

N 

205 

120 

134 

175 

120 

107 

Pseudo R 2 

0.241 

0.152 

0.215 

0.112 

0.209 

0.126 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 

'The median ELA focus list enrollment is 10 percent of the school's total enrollment. In math, the median focus list enrollment is 8 
percent of schools' total enrollment. 


Exhibit B-12 

City Year schools' likelihood of improved achievement in ELAand math from 2012-13 to 2013- 
14# compared with non-City Year schools and with City Year schools with more than the median 
percent of students on ELA and math focus lists, by grade level 


ELA 

Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

More than the median percent of 
a school's enrollment is on a focus 
list 1 

1.182 

[0.634] 

1.912 

[1.151] 

1.895 

[1.689] 

0.865 

[0.634] 

1.914 

[1.178] 

0.841 

[0.614] 

School-level percent poverty 

0.977 

[0.026] 

0.977 

[0.029] 

0.969 

[0.035] 

1.020 

[0.035] 

1.005 

[0.029] 

0.959 

[0.050] 

School-level percent ELL 

1.030 

[0.018] 

1.019 

[0.021] 

0.970 

[0.024] 

1.011 

[0.019] 

1.012 

[0.028] 

1.024 

[0.029] 

Constant 

3.274 

[8.689] 

5.569 

[16.088] 

8.627 

[28.553] 

0.403 

[1.342] 

0.449 

[1.251] 

14.597 

[66.495] 

N 

128 

96 

90 

96 

83 

97 

Pseudo R 2 

0.136 

0.0750 

0.158 

0.116 

0.0657 

0.0999 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 

'The median ELA focus list enrollment is 10 percent of the school's total enrollment. In math, the median focus list enrollment is 8 percent of 
schools' total enrollment. 
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Exhibit B-13 

City Year schools' likelihood of improved achievement in ELAand math compared with non-City 
Year schools and with City Year schools that began their partnership after 2011-12. by year 


ELA 

Math 


2010 to 

2011-12 to 

2012-13 to 

2010 to 

2011-12 to 

201 2-1 3 to 


2011-12 

2012-13 

2013-14 

2011-12 

2012-13 

2013-14 

School began City Year 

1.076 

1.104 

1.109 

1.616 

3.118* 

1.428 

partnership after 2011-12 

[0.617] 

[0.445] 

[0.482] 

[0.935] 

[1.401] 

[0.658] 

School-level percent poverty 

0.945* 

[0.022] 

1.029* 

[0.013] 

0.994 

[0.025] 

0.942 

[0.029] 

0.997 

[0.010] 

0.972 

[0.025] 

School-level percent ELL 

1.042 

[0.027] 

0.998 

[0.017] 

1.011 

[0.016] 

1.000 

[0.025] 

1.022 

[0.025] 

1.005 

[0.021] 

Constant 

276.021* 

0.195 

4.844 

453.219 

1.658 

30.896 

[684.975] 

[0.324] 

[12.745] 

[1,449.847] 

[2.507] 

[82.777] 

N 

97 

168 

143 

97 

164 

128 

Pseudo R 2 

0.171 

0.185 

0.0866 

0.167 

0.263 

0.0775 


Reported in odds ratios. Standard errors in brackets. * p < 0.05; ** p < 0.01 ; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


Exhibit B-14 

City Year schools' likelihood of improved achievement in ELAand math, by length of City Year 

partnership, by year 


ELA Math 


2011-12 to 

2012-13 to 

2011-12 to 

2012-13 to 

2012-13 

2013-14 

2012-13 

2013-14 


Length of school's City 

Year partnership: 

1 year 

1.751 

[0.710] 

2.841** 

[0.830] 

2.232 

[0.922] 

4.583*** 

[1.777] 

2 years 

4.206** 

[2.182] 

1.540 

[0.595] 

1.408 

[0.637] 

5.515*** 

[2.477] 

3 or more years 

0.965 

[0.370] 

1.938* 

[0.560] 

1.221 

[0.480] 

1.516 

[0.441] 

School-level percent poverty 

0.958*** 

[0.007] 

1.017** 

[0.006] 

0.974*** 

[0.007] 

0.996 

[0.006] 

School-level percent ELL 

1.004 

[0.007] 

1.009 

[0.007] 

0.997 

[0.008] 

1.010 

[0.008] 

Constant 

1.751 

[0.710] 

2.841** 

[0.830] 

2.232 

[0.922] 

4.583*** 

[1.777] 

N 

511 

577 

458 

535 

Pseudo R 2 

0.138 

0.153 

0.0920 

0.169 


Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 
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Exhibit B-15 

OVERALL Implementation rank, likelihood of improvement in school-level 

ELA achievement, all schools 


ELA 

High-implementing City Year partnered school 


1.835* 

[0.556] 

Low-implementing City Year partnered school 


1.952 

[0.715] 

School-level percent poverty 


0.961** 

[0.012] 

School-level percent ELL 


1.014 

[0.009] 

Constant 


10.350 

[13.66] 


N 

432 

Pseudo R 2 

0.12 


Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 


Exhibit B-16 

Math implementation, likelihood of improvement in 
school-level math achievement, City Year partner schools 



Math 

High-implementing City Year partnered school 

2.503* 

[0.024] 

School-level percent poverty 

0.967 

[0.079] 

School-level percent ELL 

1.025 

[0.084] 

Constant 

1 1 990 

[0.149] 

N 

125 

Pseudo R 2 

0.05 


Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 
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Fixed -effects Linear Regression Results 


Exhibit B-17 

City Year partnership and school-level ELAand math assessments 
(in standard deviations), all schools 



ELA 

Math 

City Year school 

0.074* [0.030] 

0.080* [0.038] 

School-level percent poverty 

-0.000 [0.001] 

-0.004 [0.002] 

School-level percent ELL 

0.001 [0.001] 

0.002 [0.002] 

Constant 

-0.015 [0.103] 

0.292 [0.254] 

N 

313 

258 

R 2 

0.231 

0.246 

Adjusted R 2 

0.192 

0.200 

Standard errors in brackets. * p < 0.05; ** 

p < 0.01 ; *** p < 0.001 



Model includes site-level fixed effects (not shown). 


Exhibit B-18 

City Year partnership and school-level 
ELA and math assessments (in standard deviations), by grade level 




ELA 


Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school 

0.107 

[0.061] 

0.115* 

[0.050] 

0.384 

[0.181] 

0.042 

[0.042] 

0.088* 

[0.035] 

0.097 

[0.062] 

School-level percent 
poverty 

-0.011** 

[0.004] 

0.005 

[0.005] 

-0.021 

[0.016] 

-0.008** 

[0.003] 

-0.001 

[0.003] 

-0.001 

[0.007] 

School-level percent ELL 

0.005 

[0.003] 

-0.002 

[0.003] 

- 0.000 

[0.005] 

0.002 

[0.002] 

-0.001 

[0.002] 

-0.001 

[0.002] 

Constant 

0.929* 

[0.393] 

-0.496 

[0.448] 

1.956 

[1.334] 

0.718** 

[0.269] 

0.106 

[0.308] 

0.563 

[0.560] 

N 

123 

107 

71 

123 

107 

70 

R 2 

0.314 

0.263 

0.452 

0.392 

0.393 

0.456 

Adjusted R 2 

0.232 

0.160 

0.270 

0.319 

0.308 

0.385 

Standard errors in brackets. 

* p < 0.05; ** 

p < 0.01 *** 

p < 0.001 





Model includes site-level fixed effects (not shown). 
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Exhibit B-19 

ELA and math focus list enrollment and school-level 
ELA and math assessments (in standard deviations), all schools 



ELA 

Math 

More than the median percent of a school's 
enrollment is on a focus list 1 

0.067 [0.037] 

0.158** [0.057] 

School-level percent poverty 

-0.000 [0.001] 

-0.004 [0.002] 

School-level percent ELL 

0.001 [0.001] 

0.002 [0.002] 

Constant 

0.003 [0.107] 

0.302 [0.260] 

N 

372 

238 

R 2 

0.224 

0.270 

Adjusted R 2 

0.184 

0.221 

Standard errors in brackets. * p < 0.05; ** p < 0.01; 

***p< 0.001 



Model includes site-level fixed effects (not shown). 

1 For schools included in our final analyses, the median ELA focus list enrollment is 10 percent of the school's 
total enrollment. In math, the median focus list enrollment is 8 percent of schools' total enrollment. 

Exhibit B-20 

ELA and math focus list enrollment and school-level 
ELA and math assessments (in standard deviations), by grade level 


ELA 

Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

More than the median percent 
of a school's enrollment is on a 
focus list 1 

0.060 

[0.049] 

0.096* 

[0.041] 

0.047 

[0.076] 

0.057 

[0.102] 

0.206** 

[0.063] 

0.287 

[0.302] 

School-level percent poverty 

-0.008** 

[0.003] 

- 0.000 

[0.003] 

- 0.000 

[0.007] 

-0.010** 

[0.004] 

0.004 

[0.005] 

-0.011 

[0.018] 

School-level percent ELL 

0.002 

[0.002] 

-0.001 

[0.002] 

-0.002 

[0.002] 

0.004 

[0.003] 

-0.003 

[0.003] 

0.000 

[0.006] 

Constant 

0.736** 

[0.270] 

0.079 

[0.319] 

0.599 

[0.584] 

0.877* 

[0.395] 

-0.490 

[0.484] 

1.308 

[1.529] 

N 

7 23 

106 

70 

709 

103 

77 

R 2 

0.395 

0.390 

0.437 

0.317 

0.303 

0.299 

Adjusted R 2 

0.322 

0.303 

0.364 

0.224 

0.202 

0.0655 


Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 

1 For schools included in our final analyses, the median ELA focus list enrollment is 10 percent of the school's total enrollment. In math, the 
median focus list enrollment is 8 percent of schools' total enrollment. 
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Exhibit B-21 

Number of corps members per school and school-level 
ELA and math assessments (in standard deviations), all schools 




ELA 

Math 

City Year school with more than 3 corps 
members for every 1 00 students 


0.114*[0.045] 

0.186**[0.056] 

School-level percent poverty 


-0.000(0.001] 

-0.005* [0.002] 

School-level percent ELL 


0.002 [0.001] 

0.002 [0.002] 

Constant 


0.035 [0.095] 

0.436 [0.257] 


N 

300 

246 


R 2 

0.205 

0.214 

Adjusted R 2 

0.163 

0.163 

Standard errors in brackets. * p < 0.05; ** 

p < 0.01; 

*** p < 0.001 



Model includes site-level fixed effects (not shown). 

1 The median ELA focus list enrollment is 10 percent of the school's total enrollment. In math, the median 
focus list enrollment is 8 percent of schools' total enrollment. 

Exhibit B-22 

Corps member to student ratio more than 3:1 00 and school-level ELA and 
Math achievement (in standard deviations), by grade level 


ELA 

Math 


Elementary 

Middle 

High 

Elementary 

Middle 

High 

City Year school with 
more than 3 corps 
members for every 100 
students 

0.056 

[0.059] 

0.117* 

[0.051] 

0.289** 

[0.103] 

0.149 

[0.088] 

0.196** 

[0.073] 

0.619*** 

[0.100] 

School-level percent 
poverty 

-0.008** 

[0.002] 

- 0.000 

[0.003] 

-0.007 

[0.006] 

-0.010** 

[0.004] 

0.005 

[0.005] 

-0.030** 

[0.006] 

School-level percent ELL 

0.001 

[0.002] 

-0.001 

[0.002] 

- 0.000 

[0.002] 

0.002 

[0.003] 

-0.003 

[0.003] 

0.000 

[0.002] 

Constant 

0.662* 

[0.259] 

0.111 

[0.313] 

0.913 

[0.540] 

0.834* 

[0.388] 

-0.499 

[0.447] 

2.524** 

[0.528] 

N 

779 

7 06 

63 

779 

7 06 

47 

R 2 

0.413 

0.385 

0.512 

0.313 

0.280 

0.944 

Adjusted R 2 

0.340 

0.299 

0.440 

0.228 

0.178 

0.907 


Standard errors in brackets. * p < 0.05; ** p < 0.01; *** p < 0.001 
Model includes site-level fixed effects (not shown). 
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APPENDIX C 

Coding Method for Binary Outcome Variables 
Used in the Logistic Regressions 
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We used states' ELA and math assessment results to create two binary outcome variables, indicating whether a school's 
performance improved in ELA and math from the prior year (i.e., one binary variable for ELA and one for math). To ensure 
the greatest comparability across states, we developed a two-pronged coding method that would permit us to create 
binary outcome variables for both states that report school-level scaled scores and states that only publicly report school- 
level proficiency rates. The following discussion describes both methods. 

Grade-level Mean Scaled Scores 

For the 13 City Year sites in states that release scale score data, we used the following process to calculate the binary 
outcome variables: 

1 . We generated standardized scores for each grade using grade-level scaled score data for each school, the 
reported statewide grade-level scaled score, and standard deviation data for each grade. 

2. We created a school-level standardized score by taking a weighted average of standardized scores across grades. 
We weighted by the reported number of tests scored at each grade to ensure that, for instance, in a school with 
20 third-graders and 80 fourth-graders, the proportion of the school enrolled in third-grade would be reflected 
in the aggregate school standardized score. 

3. Once we calculated the aggregate school standardized score, for each CY and comparison school and for each 
year for which we had data, we took the difference between the school's mean standardized score and the 
school's mean standardized score in the prior year. We coded the outcome variable as "0" for schools for which 
the result of this calculation was zero, or negative. We coded the outcome variable as "1 " for schools for which 
the result was positive. 

School-level Percent Proficient 

For the nine City Year sites in states for which we had only school-level percent proficient data, we calculated the change in 
the percent of students who had met the state's proficiency target compared with the prior year. Specifically, this measure 
was simply whether a greater percent of a school's students had scored above their state's proficiency target; we did not 
examine students' movement within the categories that comprise above or below proficient (e.g., below basic, basic, 
advanced). For example, if a greater percent of a school's population scored at or above ELA proficiency over the prior 
year, and the gain was greater than one percent, the outcome variable for ELA was coded as "1 ". If, however, the percent of 
students scoring above proficient on the state's ELA assessment declined over the prior year or was less than one percent, 
we coded the value as "0". 
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